Judging the judges: Evaluating the performance of international gymnastics judges

Judging a gymnastics routine is a noisy process, and the performance of judges varies widely. The International Federation of Gymnastics (FIG), in collaboration with Longines and the Université de Neuchâtel, is designing and implementing an improved statistical engine to analyze the performance of gymnastics judges during and after major competitions like the Olympic Games and the World Championships. The engine, called the Judge Evaluation program (JEP), has three objectives: (1) provide constructive feedback to judges, executive committees and national federations; (2) assign the best judges to the most important competitions; and (3) detect bias and outright cheating. In this article, using data from international competitions held during the 2013-2016 Olympic cycle, we first develop a marking score evaluating the accuracy of the marks given by judges. We then study ranking scores assessing to what extent judges rate gymnasts in the correct order, and explain why we ultimately chose not to implement them. We study outlier detection to pinpoint athletes that were poorly evaluated by judges. Finally, we discuss interesting observations and discoveries that led to recommendations to the FIG.